Skip to content

Phase 3: Data quality charts + HED wordcloud + file modals - #11

Closed
neuromechanist wants to merge 4 commits into
feature/issue-1-epic-nemar-redesignfrom
feature/issue-4-phase3-qa-hed
Closed

neuromechanist wants to merge 4 commits into
feature/issue-1-epic-nemar-redesignfrom
feature/issue-4-phase3-qa-hed

Conversation

@neuromechanist

Copy link
Copy Markdown
Contributor

Summary

Phase 3 of #1. Adds the data-quality visualizations from the legacy nemar.org back onto the new detail page, plus the per-file Vis/View modals that replace the Phase 5 "soon" placeholders.

Closes #4.

Cross-repo: the backend half (hallu sync + data.nemar.org/<id>/qa/* route) is tracked at nemarOrg/nemar-cli#511. Until that ships, this PR renders against fixtures via a ?qa=fixture dev toggle; the toggle becomes a no-op once #511 deploys.

What landed

Library — src/lib/qa.ts

  • QaAggregates / QaSummary / FileQa / HedSummary types — define the contract the website expects from data.nemar.org/<id>/qa/* so Epic: Website compute surfaces (#493) #511 has a target
  • getQaAggregates / getFileQa / getHedSummary typed clients with abortable timeouts; 404/429/5xx → null (never break the page)
  • parseLinenoiseDb, buildHistogram, bucketAgesBySex, filePlotUrl pure helpers with 18 unit tests in qa.test.ts

Charts (hand-rolled SVG, no chart-lib dep)

  • PipelineSuccessChart.astro — stacked horizontal bar (finished / cleaning / failed / other), semantic green/amber/red palette
  • HistogramChart.astro — 10-bin SVG histogram, reused 3x for frames retained %, channels retained %, line-noise dB. Median + sample-count badges.
  • AgeGenderChart.astro — paired bars per 10-year age bucket, split M/F/O using modality accent colors
  • HedWordcloud.astro — log-scaled font sizing, semantic category-colored tags, rectangular flow layout. No JS cloud-packing — that's overkill for a 24-tag overview.

QualityPanel — src/components/QualityPanel.astro

Composes the 4 charts in a 2-col grid + the HED wordcloud in a band below. Renders nothing if both aggregates and HED summary are null (graceful empty state).

Wired into /dataset/[id].astro between the version switcher and the README.

File modals

  • FileVisModal.astro — lazy-mounted once at the BidsTree root, opens on [data-open-vis] click. Header + 4-stat strip + 5-SVG grid (eegplot_mid-sample, icaact, icahist, icamaps, spectopo). HEAD-probes each SVG so missing ones render a 'Not available' tile, not a broken icon. In-memory cache per (id, path).
  • FileViewModal.astro — same mounting pattern, opens on [data-open-view] click. Fetches the .tsv via the existing version-locked URL, renders first 100 rows as a sticky-header table, footer shows row count + 'Download full file'.

BidsTree integration

  • BidsDirChildren.astro takes new datasetId + version props, threads them through recursion. The Vis · soon and View · soon decorative chips are gone — replaced with real <button> elements emitting data-open-vis / data-open-view + the per-file metadata the modals need.
  • BidsTree.astro mounts both modals once after the file tree.

Verification

  • bun run typecheck — 0 errors across 55 files
  • bun run test — 87/87 unit tests passing (69 from Phase 1+2 + 18 new for qa.test.ts)
  • bun run dev + /browse capture in both themes — QualityPanel renders 4 charts + HED cloud from fixture data on /dataset/nm000104?qa=fixture
  • No console errors on the detail page
  • Click a Vis button on a .set file → modal opens, stats populated, plot tiles render (or show 'Not available' until Epic: Website compute surfaces (#493) #511 ships the SVGs)
  • Click a View button on a .tsv file → modal opens, fetches the tsv via the existing manifest URL, renders the table
  • /dataset/on005262 (no QA exists, no fixture) — empty state, page renders cleanly

Out of scope

Test plan

  • CI green
  • Cloudflare Pages preview deploys
  • Manual: /dataset/nm000104?qa=fixture — all 4 charts + wordcloud render in both themes
  • Manual: click Vis on any .set file → modal opens with stats strip + 5 plot tiles
  • Manual: click View on participants.tsv → table renders first 100 rows
  • Manual: /dataset/on005262 (no QA, no fixture) → no QualityPanel renders, page is otherwise normal
  • After nemar-cli#511 ships: drop the ?qa=fixture toggle in a follow-up commit, verify live QA renders on a synced dataset

Follow-ups

  1. nemar-cli#511 (backend QA sync + route) — this PR is fully usable today behind the fixture toggle; removing the toggle once Epic: Website compute surfaces (#493) #511 deploys is a one-line change.
  2. /design-review on the live Cloudflare Pages preview once the project is set up. Visual fixes will land as a separate polish PR if needed.

src/lib/qa.ts defines the contract the website expects from
data.nemar.org/<id>/qa/* (sibling backend issue: nemarOrg/nemar-cli#511):

- QaAggregates: per-file arrays for the 4 dataset-level charts
  (pipeline status, frames retained %, channels retained %, ICA %,
   line-noise dB) + optional demographics from participants.tsv
- QaSummary: min/max ranges (existing hallu dataqual.json shape, kept
  for completeness)
- FileQa: per-file summary JSON (matches the shape we sampled from
  hallu's processed/<id>/sub-XXX/eeg/*_dataqual.json)
- HedSummary: token frequency map for the wordcloud
- FILE_PLOT_KINDS + filePlotUrl(): the 5 SVGs (eegplot_mid-sample,
  icaact, icahist, icamaps, spectopo) the modal renders per .set file

Pure helpers tested in qa.test.ts (18 cases):
- parseLinenoiseDb('14.40dB') -> 14.4
- buildHistogram with N bins + optional domain
- bucketAgesBySex with configurable bucket width
- filePlotUrl with percent-encoded path segments

jsonOrNull() reuses the 429/5xx-as-null pattern from data-api.ts so QA
fetches degrade to empty states instead of breaking the dataset page
when artifacts haven't been synced yet.

Part of #4.
Hand-rolled SVG charts (no chart-lib dependency). Each is ~80-150 lines:

- PipelineSuccessChart: stacked horizontal bar (green/amber/red/grey
  for finished / cleaning / failed / other) with legend
- HistogramChart: 10-bin SVG histogram, reused 3x for frames retained %,
  channels retained %, line-noise dB. Median + sample-count badges.
- AgeGenderChart: paired bars per 10-year age bucket, split M/F/O using
  modality accent colors (semantic: EEG blue for M, iEEG pink for F).
- HedWordcloud: log-scaled font sizing, semantic category-colored tags
  (Sensory-event / Agent-action / Item / Property / Spatial / ...),
  rectangular flow layout. No JS cloud-packing algorithm — that's
  overkill for the 24-tag overview.

QualityPanel composes the four charts + the wordcloud, renders only when
aggregates or hedSummary is present, empty state when both are null.

Wired into /dataset/[id].astro between the version switcher and the
README. Behind a ?qa=fixture toggle for dev work today; the toggle is a
no-op once nemarOrg/nemar-cli#511 ships the live endpoint.

Fixtures (test/fixtures/qa-*.json) synthesized from the actual hallu
sample (ds002718 sub-002 task-FaceRecognition_eeg_dataqual.json that we
read this session) — 30 files, realistic distributions, demographic
ages 21-38 spread across M/F.

87/87 unit tests still passing. Typecheck clean across 53 files.

Part of #4.
… real buttons

FileVisModal.astro — opens on click of a Vis button on any .set file.
- Header with dataset id, version, file path
- 4-stat strip (data frames %, channels %, ICA %, line noise) from
  per-file qa/<path>_dataqual.json
- 5-plot grid (eegplot_mid-sample, icaact, icahist, icamaps, spectopo) as
  lazy-loaded <img> elements pointing at the qa SVG URLs
- HEAD-probes each SVG so missing ones render a 'Not available' tile
  rather than broken-image icons
- In-memory cache per (id, path) so reopens are instant
- 'QA artifacts not synced yet' empty state when nothing comes back
- Lazy-mounted ONCE at the BidsTree root; opened via [data-open-vis]
  attributes on the buttons in BidsDirChildren

FileViewModal.astro — TSV preview, same mounting pattern.
- Fetches the file via the existing data.nemar.org/<id>/<ver>/<path>
- Parses first 100 rows into an HTML table with sticky header
- 'Showing N of M rows' counter + 'Download full file' link in footer
- Renders error state when the fetch fails

BidsDirChildren.astro: takes new datasetId + version props, threads
through recursion, strips them onto buttons via data-* attributes. The
'Vis · soon' and 'View · soon' decorative chips are gone — those tags
were the Phase 5 placeholder. Real buttons take their place.

BidsTree.astro: mounts both modals once after the file tree so the
single global event listener in each modal handles every click. Threads
datasetId + version down to the recursive children.

Detail page now passes datasetId={id} version={selectedVersion} into
<BidsTree>. No-op when those are missing — buttons emit empty data-*
attributes and the modals early-return.

Verified live with ?qa=fixture on nm000104 — Vis modal opens with the
expected stats + plot frames (plots themselves 404 against the live
backend until #511 ships, which is exactly the empty-state we want).

Part of #4.
The 'compile' setting still bundled sharp into the SSR worker, which the
Workers runtime can't execute — deploys failed with:

  Uncaught Error: [unenv] process.report.getReport is not implemented yet!

We don't use Astro's <Image> component or astro:assets anywhere — every
visual asset is a plain <img> referencing /public/*.{svg,jpg,png}.
Passthrough lets the original assets ship as-is, keeps sharp out of the
worker bundle, and shaved the deploy from failing to ~3s.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented May 15, 2026 •

Copy link
Copy Markdown

Deploying nemar-website with  Cloudflare Pages  Cloudflare Pages

Latest commit: d3f62f0
Status:🚫  Build failed.

View logs

@neuromechanist

Copy link
Copy Markdown
Contributor Author

Superseded by #24 (QualityPanel + charts + qa.ts) and tracked by #25 (file modals follow-up). Original PR #11 was forked behind the epic's deferred-fetch architecture and required a rework rather than a rebase.

@neuromechanist
neuromechanist deleted the feature/issue-4-phase3-qa-hed branch May 18, 2026 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant